A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Second-order optimization methods have the ability to accelerate convergence by modifying gradient through curvature matrix. There been many attempts use second-order for training deep neural networks. In this work, inspired diagonal approximations and factored such as Kronecker-factored Approximate Curvature (KFAC), we propose a new approximation Fisher information matrix (FIM) called Trace-restricted (TKFAC), which can hold certain trace relationship between exact approximate FIM. TKFAC, decompose each block of FIM Kronecker product two smaller matrices scaled coefficient related trace. We theoretically analyze TKFAC's error give an upper bound it. also damping technique TKFAC on convolutional networks maintain superiority during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms some network architectures.